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Abstract;  The  general  theory  of  parameter  estimation  is  developed  using  the 
inverse  probability  approach.  Where  the  measurements  are  perturbed  by 
additive  Gaussian  noise  and  when  the  received  information  is  sufficient  to 
determine  the  parameters  of  interest  rather  accurately,  it  is  shown  that  an 
optimum  method  of  processing  redundant  data  based  on  the  maximum  likeli¬ 
hood  approach  reduces  approximately  to  the  solution  of  k  nonlinear  equations 
in  the  k  unknown  parameters.  An  expression  is  derived  for  the  resulting  error 
moment  matrix  of  the  parameters.  It  is  shown  that  this  same  moment  matrix 
for  a^  minimum  variance  estimate  is  obtained  by  using  results  derived  by 
Cramer.  The  theory  is  illustrated  by  applying  it  to  several  radar  measurement 
problems  of  interest. 
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PARAMETER  ESTIMATION  THEORY  AND  SOME  APPLICATIONS 
OF  THE  THEORY  TO  RADAR  MEASUREMENTS 


THE  METHOD  OF  INVERSE  PROBABILITY* 

Suppose  we  have  some  system  about  which  we  are  to  receive  information  and  that  the 
state  of  this  system  is  characterised  by  a  set  of  k  parameters.  We  receive  a  message  or 
make  measurements  on  the  system,  which  are  generally  perturbed  by  noise,  and  on  the  basis 
of  this  information  and  our  own  a  priori  information,  we  must  arrive  at  some  optimum  estimate 
about  the  state  of  the  system. 

We  make  the  following  definitions: 

y  =  [yii  y2>  ■  •  • .  yul  =  t™e  parameters 

y  =  [yi.  y2>  •  •  • .  yj  =  estimated  parameter  values 

X  =  X(y)  =  [xi,  X2,  •  •  • ,  x„]  =  noiseless  message 

Y  =  [yi,  y2i  ■  *  • ,  y J  =  message  perturbed  by  noise 

The  state  of  the  system  is  thus  characterized  by  a  point  or  vector  in  a  k-dimensional  param¬ 
eter  space.  The  quantities  y,  y,  X,  and  Y  will  denote  column  vectors,  though  vector  signs 
have  been  omitted  for  simplicity  of  notation.  A  bar  will  be  used  to  denote  average  or  ex¬ 
pected  value,  and  t  will  denote  transpose. 

In  the  situation  where  the  number  of  measurements  n  is  less  than  the  number  of  param¬ 
eters  k,  a  noiseless  message  X  is  not  sufficient  to  determine  y  uniquely.  When  n=k,  the 
message  X  is  in  general  just  sufficient  to  determine  y  and  we  have  the  so-called  minimum 
data  case.  When  n>k,  y  is  in  general  over-specified  by  X  and  we  have  the  so-called  re¬ 
dundant  data  case. 

The  received  message  or  measurement  Y  differs  from  X  because  of  noise,  and  the  effect 
of  this  noise  is  characterized  by  the  conditional  probability  density  P(Y/X)  [or  P(Y/y)  since 
X  =  X(y)l.  Given  the  message  Y  we  can  express  the  a  posteriori  probability  density  over  y 
in  terms  of  P(Y/y)  and  the  a  priori  probability  density  P(y)  with  the  aid  of  Bayes’  theorem. 

P(y/Y)  =  P(Y/y)P(y) 

P(Y) 

where  P(Y)  =  Jp(Y/y)P(y)dy  (1) 

and  J[  ]dy  stands  forj  •  ^  •  J[  ]dyi  dy2  •  •  •  dyjj 
*  Molt  of  the  theory  of  thli  paper  followi  the  material  preiented  in  Reference  1. 


Since  we  ate  interested  in  the  dependence  of  P(y/Y)  on  y,  we  shall  find  it  convenient  to 
ignore  factors  which  are  independent  of  y  and  write 

P(y/Y)-P(Y/y)P(y)  (2) 

where  ~  stands  for  “is  proportional  to” 

It  is  convenient  to  carry  through  calculations  on  probability  densities  in  terms  of  proportion¬ 
alities,  recognizing  that  the  constant  of  proportionality  can  always  be  inserted  later  simply 
by  requiring  that  the  integrated  probability  be  equal  to  unity. 

The  above  formulas  contain,  in  principle,  all  one  needs  to  know  about  the  parameter 
estimation  problem.  The  value  of  y  which  maximizes  P(Y/y)  is  called  the  maximum  likeli¬ 
hood  estimate.  When  P(y)  can  be  considered  to  be  constant  over  the  region  of  interest,  the 
maximum  likelihood  estimate  also  maximizes  P(y/Y).  We  ..proceed  now  to  specialize  the 
theory  to  the  case  of  additive  Gaussian  noise. 

SPECIALIZATION  OF  THE  THEORY  TO  THE  CASE  OF  ADDITIVE  GAUSSIAN  NOISE 

Assume  that  the  error  Y-X  results  from  additive  Gaussian  noise  characterized  by  an 
n  X  n  moment  matrix  M.  Then,  P(Y/y)  is  a  multidimensional  normal  distribution  of  the  form* 

P(Y/y)=P[Y/X(y)]  =  P[Y-X(y)]~e-^-[^-^(>'>]’'  [Y-x(y)] 

_  (3) 

where  M  =  [Y-X(y)]  [Y-X(y)]'f 

Substituting  this  expression  for  P(Y/y)  into  Eq.  2,  we  obtain 

P(y/Y)~  P(Y/y)P(y)~  P(y)e-y'tY-x(y)]’'  m-i  [v-xey)] 

If  the  measurements  are  capable  of  leading  to  a  rather  accurate  determination  of  y,  the  argu¬ 
ment  of  the  exponential  will  be  sharply  peaked  around  the  correct  value.  If  P(y)  is  a  very 
slowly  varying  function  of  y  over  the  region  of  interest,  we  may  justifiably  treat  it  as  a  con¬ 
stant  of  proportionality.  With  this  assumption,  the  estimate  y  which  maximizes  P(y/Y)  also 
maximizes  KY/y)  and  is  therefore  a  maximum  likelihood  estimate.  In  order  for  y  to  be  a 
maximum  likelihood  estimate,  it  must  satisfy  the  vector  equation 

|‘/r[Y-X(y)]'fM-‘[Y-X(y)]|=-[^^j^M-‘[Y-X(y)]  =  0  (5) 

where  it  is  understood  that is  an  n  x  k  matrix  with  ij*'*  element  .  The  solution 
for  y  is  seen  to  involve  the  solution  of  k  generally  nonlinear  equations  in  the  k  unknown 
parameters. 


*  Fora  diicuBBion  of  the  multidimefiBionsl  normal  distribution  see,  for  example,  Reference 


Situations  can  arise  in  which  more  than  one  value  of  y  may  satisfy  these  equations. 
Some  of  the  values  represent  local  maxima  or  saddle  points,  or  perhaps  even  local  minima 
of  Pfy/Y).  In  these  situations  we  must  find  other  means  for  deciding  which  one  of  the  values 
of  Y  corresponds  to  an  absolute  maximum  of  P(y/Y). 


Using  the  earlier  assumption  that  Y  leads  to  a  rather  accurate  determination  of  y,  and 
assuming  that  the  functional  dependence  of  X  on  y  is  sufficiently  well  behaved,  we  can 
approximate  in  the  region  of  small  errors 


X(y)  =  X(y)  + 


^X(y) 


dy 

where  dy  =  y  -  y 


dy 


(6) 


where  higher  order  terms  have  been  neglected.  Using  this  expression  in  Eq.  4  for  P(y/Y), 
we  have 


The  first  terra  in  the  exponent  of  e  does  not  depend  on  y  and  therefore  represents  just  a  con¬ 
stant  of  proportionality.  The  second  term  in  the  exponent  is  zero  by  virtue  of  Eq.  5.  Eq.  7 
then  reduces  to 

p(y/Y)  ~  ^  ^ 


Our  assumptions  have  led  us  to  a  P(y/Y)  which  is  a  multidimensional  normal  distribution 
with  a  k  X  k  moment  matrix  F.  The  variance  in  the  i***  parameter  is  Fj*.  In  order  that  P(y/Y) 
be  a  relative  maximum,  it  is  necessary  that  the  quadratic  form  dy^F“'  dy  be  always  positive 
for  dy  ^  0.  It  follows  that  the  matrix  F”^  (and  also  F)  must  be  positive  definite.* 

It  is  interesting  to  compare  the  result  obtained  here  for  the  error  moment  matrix  of  the 
parameters  with  that  obtained  by  Cramer.  Cramer  (Reference  2)  has  shown  that  a  joint  ef¬ 
ficient  minimum  variance  estimate  has  an  error  moment  matrix  F  which  satisfies 

*  For  fi  discussion  of  positive  definiteness,  see,  for  example,  p.  273  of  Reference  3. 


3 


4 


(f-*)  ^  log  P(Y/y)  ^  log  P(Y/y) 

where,  as  before,  the  bar  denotes  an  average.  Inserting  P(Y/y)  from  Eq.  3  into  this  ex¬ 
pression,  we  have 


dlogm/y)  ^  p^)j'‘  „-i  [Y  _  x(y)]  =  [Y  _  x(y)]’‘  M-‘  ^  (10) 


Then  (r-%  =  p|^j’^  M-'  [ Y  -  X(y)]  [  Y  -  X(y)]^ 

=  M-'  [Y  -  X(y)]  [Y  -  X'^)'F  M-^  ^ 

(11) 

L  f^yi  J 


If  we  take  for  the  value  of  y  in  this  equation  the  best  estimate  y,  we  see  that  this  expression 
for  r  agrees  with  that  obtained  earlier  (Eq.  8). 


i 


EXTENSION  OF  THE  THEORY  TO  INCLUDE  A  PRIORI  INFORMATION  ABOUT 
THE  PARAMETERS 

Suppose  that  our  a  priori  knowledge  about  the  parameters  may  be  approximately  charac¬ 
terized  by  a  multidimensional  normal  probability  distribution. 

P(y)~e-'^(>'->'*)’^  (y-y*)  (12) 

where  9  is  the  moment  matrix  of  the  distribution  and  y*  =  [y*,  y*,  •  •  • ,  y*]  is  the  most 
probable  a  priori  value  of  y.  Then 

P(y/Y)~  P(y)P(Y/y)-  e-y’<y''y>'^  (y-y*)  -y. [v- x(y)]'^  m-J  [Y-x(y)]  (13) 


If  y  is  that  value  of  y  which  maximizes  P(y/Y),  y  must  satisfy 
d 


‘/2(y-y*)'^!p->  (y-y*)  +  H[Y-X(y)]TM-MY-X(y)]  =0  (14) 


This  leads  to  a  generalization  of  Eq.  5  which  y  must  satisfy. 


'■iy:0.aa.iu£'.'.-  . '?.ou;aLMuiiir7;2cxits>:i 


unHuiw.gnal.i'iu 


when  y  =  y 


T 

M-‘[Y-X(y)]  =  0 


Once  again  the  optimum  processing  of  received  data  to  determine  y  involves  the  solution 
of  k  nonlinear  equations  in  tb.e  k  unknown  parameters.  As  before,  we  approximate 

X(y)=X(y)  +  ^|.  dy.  dy  =  y-y  (16) 

Substituting  this  approximation  into  Eq.  13,  using  Eq.  15,  and  ignoring  constants  of  pro¬ 
portionality,  we  have  a  result  analogous  to  Eq.  8 

P(y/Y)~e-‘''“‘>'^  dy 


where  T" '=!?-'  +  M"*  ^  . 

L  (fy  Jy  ^  y 

(r-)„  =  [Mm-®  I 


SPECIALIZATION  OF  THE  THEORY  TO  DIAGONAL  M 

Let  us  consider  briefly  the  situation  which  occurs  very  frequently  in  which  the  error 
in  each  measurement  is  statistically  independent  of  the  errors  in  the  other  measurements. 
Denoting  the  variance  of  the  error  in  the  u**"  measurement  by  M  has  the  following 
diagonal  form 


>  1 


M"*  = 


Equation  5  for  y  then  reduces  to 


y  ^  (Yu  -  X„)  ^ 

u=i 


(i  =  l,2,...,k) 


Equation  8  for  F  reduces  to 


V  dXu  1  ^X„ 

U  /ij=  A  -T - 2  ^ 

U=1  dyi  0^  dyj 


SPECIFIC  APPLICATIONS  OF  THE  THEORY 

Norrowband  Signal  With  Unicnawn  Amplitud*  and  Corritr  Phast  in  th»  Prtstnct  of  Additive 
Gaussian  Noise 

Following  the  appropriate  narrowband  filtering  operation  on  a  narrowband  signal  im¬ 
bedded  in  wideband  Gaussian  noise,  the  output  at  a  particular  moment  can  be  resolved  into 
quadrature  components  Yi  and  Yz  given  by 

Y I  =  A  cos  ^  +  n  1 

(21) 

Y2  =  A  sin  ^  +  n2 

where  A  represents  the  unknown  amplitude  of  the  output,  (f)  the  unknown  carrier  phase,  and 
ni  and  nj  the  independent  noise  outputs  of  the  quadrature  channels  each  with  variance 
(Reference  4).  The  noise  components  ni  and  n2  have  a  two-dimensional  normal  distribution 
with  diagonal  moment  matrix  M. 


a* 

M  = 

.0 


We  identify  Xi(y)  =  y\  cos  y2,  X2(y)  =  yi  sin  yi,  where  A  =  yi  and  9^  =  y2.  T  ^  can  be 
obtained  from  Eq.  20, 


r-‘  .4-' 


/dx,y  (dXzV  ^  dXj  dXj 
\^i)  ^Wyi/  dyi  dyi  ^  dyi  dyi 

^  ^  ^  ^  (dXj'' 

dyi  dyi""  dyi  dyi  [dyi)  ^  \^2/ 


from  which  it  follow's 


1  d^d^ 

2  ^Yl  ^2  W2/  \^2/ 


We  then  have  for  5^  5^,  the  rms  errors  in  estimating  A  and  (/>,  respectively 


5a  =r,7  =a 
^  11 


=  cr/A  (radians) 


Because  F  is  diagonal,  the  errors  in  A  and  ate  uncoupled.  Note  that  the  assumption  of 
small  errors  which  led  to  Eq.  6  restricts  the  validity  of  the  expression  for  5^  to  the  case 
where  (t<<  A. 


The  quantity  k^/o'^  is  the  (peak)  signal-to-noise  ratio  after  the  narrowband  filter.  For 
matched  filter  detection  of  a  pulsed  signal  immersed  in  white  Gaussian  noise,  k^/a^  is 
given  by 


a! 

~  No 


(26) 


where  E  is  the  signal  energy  and  No  is  the  noise  power  per  unit  bandwidth  at  the  input  to 
the  matched  filter  (see,  for  example,  Reference  5).  5^,  expressed  in  terms  of  2E/No,  is 

(27) 

^  V2e7n; 

For  a  radar  pulse  with  wavelength  A,  the  rms  error  d(j)  translates  into  an  rms  error  in 
measuring  range,  d^,  given  by 


drr  4i7\/2E/No 


(28) 


This  range  error  is  usually  extremely  small  because  it  corresponds  to  a  range  error  of  only 
a  fraction  of  a  wa*  .'length.  Taken  by  itself,  this  measurement  of  range  based  on  a  measure¬ 
ment  of  carrier  phase  is  highly  ambiguous  because  0  is  measured  moduio  2n.  Situations 
exist,  however,  where  it  is  possible  to  combine  several  ambiguous  but  accurate  measure¬ 
ments  of  range  in  order  to  obtain  a  nonambiguous  determination  of  velocity  or  acceleration. 


It  is  interesting  to  compare  the  above  expression  for  range  accuracy  with  a  range  accu¬ 
racy  calculation  based  on  Woodward’s  formula  (Reference  6).  With  approximations  similar  to 
the  ones  which  have  been  made  in  this  paper.  Woodward  has  derived  the  following  general 
formula  for  rms  range  accuracy: 


^  ^ 

^"2  '  “  2/3s/2e7n; 


(29) 


where  S,  =  rms  range  error 
S,  =  rms  time  error 
c  =  velocity  of  light 
^  =  2n  (rms  bandwidth  of  signal) 


This  formula  is  usually  used  to  calculate  “coarse”  range  accuracy  available  from  a  radar 
pulse  by  inserting  into  the  formula  a  /3  which  is  calculated  by  taking  the  mean  or  carrier 
frequency  as  reference.  As  Woodward  shows,  when  the  bandwidth  is  calculated  with  zero 
frequency  as  reference,  we  obtain  a  range  accuracy  corresponding  to  the  “fine  structure” 
information  in  the  waveform.  For  a  very  narrowband  signal  with  carrier  frequency  I,  ft  ~  27rf. 
When  this  value  of  j3  is  inserted  into  Woodward’s  formula  (Eq.  29),  we  obtain  exactly  the 
same  expression  for  range  accuracy  as  that  obtained  earlier  (Eq.  28),  as  expected. 


Btit  Fit  of  a  2n(l>Dtgre«  Polynomial  to  o  Sot  of  Indopondont,  Equally  Spaced  Meosuromtnts 
With  Equal  Variance 

We  shall  take  the  number  of  measurements  n  to  be  odd  so  that  we  can  set  n  =  2N  +  1. 
(The  case  of  an  even  number  of  measurements  is  not  materially  different.)  The  time  origin 
is  taken  at  the  center  of  the  measurement  interval.  Each  of  the  measurement  errors  is  inde¬ 
pendent  and  with  variance  a^. 


2N  +  1  =  total  number  of  measurements 

to  =  time  interval  between  measurements 

=  yi  +  y2  ’•ito  +  ya  u^t,  =  noiseless  measurement  at  time  utg 
Yu  =  measurement  at  time  uto  with  noise 
<7  =  variance  of  the  error  in  each  measurement 
yi.y2,y3  =  three  unknown  signal  parameters 


.  This  moment  matrix  has  its  zeros  placed  so  that  there  is  no  coupling  between  errors  in 
the  coefficients  of  the  even  and  odd  terms  of  the  polynomial  x(t)  =  yi  +  y2t  H-yst^.  It  follows 
that  the  rms  error  in  estimating  ya  does  not  depend  on  whether  yi  or  ys  is  known,  but  the 
error  in  yj  does  depend  on  whether  y,  is  known,  and  conversely.  A  similar  argument  applies 
also  for  polynomials  of  higher  degree.  The  kv'-  error  in  yi,  y2,  ys,  denoted  by  5yj, 
and  Syj,  are  given  by 


=  r 

1 1 


3a 


^  _  r  Vi 

TZ  -  '  22 


2^/3”  a 


33 


6\/5  a 


(38) 


We  shall  use  these  formulas  later  to  derive  expressions  for  the  rms  error  in  estimating  radial 
velocity,  radial  acceleration,  and  angular  rate  from  radar  measurements. 


It  is  interesting  to  compare  Sy^  with  the  value  which  it  would  have  had  if  ya  were  a 
known  constant,  5yJ  (yj  known).  This  variance  is  easily  calculated  by  ignoring  all  rows  and 
columns  in  F”’  except  the  fii.^t  row  and  column.  This  calculation  gives  simply 


and 


In  a  similar  manner 


(y3  known)  = 


1 

(r-')n 


n 


^  gy?  _  9 

(ys  known)  4 


Sy|  (yi  known)  4 


(39) 

(40) 


(41) 


As  one  would  expect,  the  introduction  of  unknown  parameters  degrades  the  measurement 
accuracy  in  both  cases. 


Accuracy  of  Measuring  Radial  Velocity  and  Radiol  Acceleration  From  Fine  Range 
Measurements 

We  assume  that  during  a  short  interval  of  time  the  range  as  a  function  of  time  can 
be  written 

r(t)  =  to  +  vt  +  '/sat^  (42) 

where  the  constants  v  and  a  are  to  be  determined,  ro  is  assumed  to  be  unknown  also.  The 
radar  obtains  on  each  pulse  a  fine  range  accuracy  given  by  Eq.  28 


where  E  is  the  returned  signal  energy  on  one  pulse.  It  is  assumed  that  a  series  of  reflected 
radar  pulses  results  in  a  series  of  n(n>>l)  independent  range  measurements  over  a  time 
interval  T,  each  with  rms  error  a  =  B„  and  with  uniform  spacing  to.  We  have  to  =  T/(n  -1) 
=  T/n.  Combining  Eqs.  38  and  43,  we  have* 

S  _  2s/36,  _  V3~A  _  .276A 

''  n^-^^to  2nT\/2nE/No  Tv/2nE/No  ^  ^ 


.  „  12V55,  3v/rX 


ly/rx  2.14A 
V2nE/No  ~  TV2nE/No 


It  is  important  to  keep  in  mind  that  these  formulas  do  not  account  for  possible  ambiguities 
in  the  parameter  estimates. 

Accuracy  of  Meosuritig  Angular  Rote 

We  assume  that  the  angle-of-arrival,  0,  may  be  approximated  by 

0  =  0o  +  w  t  (46) 

over  a  short  measurement  interval  of  interest,  where  ©o  and  a  are  unknown  constants  which 
are  to  be  determined.  The  general  formula  for  the  maximum  angular  accuracy  in  estimating 
angle  of  arrival  is** 

where  £  =  2n-  X  rms  length  of  the  antenna(taken  about  the  centroid)and  E  and  No  are  defined 
as  before.  We  assume,  as  before,  that  we  have  a  series  of  n(n>>l)  independent,  equally 
spaced  measurements  of  angle  over  a  time  interval  T  =  nt,.  Combining  Eqs.  38  and  47, 
we  have 


g  ^  2v/3g0  2v/3'A 


3.46A 


n  to  £Ts/2hE/No  £TV2nE/No 

For  the  case  in  which  the  antenna  aperture  consists  of  two  disjoint  coplanar  apertures  with 
equal  areas  and  separated  by  a  distance  D,  which  is  large  compared  to  the  linear  dimensions 

*  The  expreiiion  agrees  with  that  derived  in  Reference  7. 

**The  juitlfication  of  this  formula  la  Indicated  in  Reference  S. 


of  each  aperture,  we  have  £  ~  ttD  and 
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(49) 


As  before,  we  must  emphasize  that  this  formula  does  not  take  into  account  the  possibility  of 
ambiguities. in  the  estimation  of  angular  rate. 


12 


¥ 

n 

K' 

% 


'K- 
^  I 


REFERENCES 


1.  R.  Mar.asse,  “The  Theory  of  Parameter  Estimation,  Processing  of  Redundant  Data  and 
the  Calculation  of  Errors,”  Memo  of  10  January,  1955  (Reissued  as  M.I.T.  Lincoln 
Laboratory  GR312-2S).  For  a  similar  treatment  see  also  R.  K.  Bennet,  H.  M.  Jones, 
I.  I.  Shapiro,  “Error  Analysis  of  a  Statistical  Data  Handling  System,”  M.I.T.  Lincoln 
Laboratory  GR312-1,  1  September,  1955.  Also  see  I.  Shapiro,  “The  Prediction  of  Bal¬ 
listic  Missile  Trajectories,”  Ch,  II  (McGraw-Hill,  1957). 

2.  H.  Cramer,  “Mathematical  Methods  of  Statistics,”  section  32.7,  page  495  (Princeton 
University  Press,  1945). 

3.  G.  Birkhoff  and  S.  MacLane,  “A  Survey  of  Modern  Algebra”  (MacMillan  Publishing  Co., 
New  York,  1941). 

4.  S.  0.  Rice,  “Mathematical  Analysis  of  Random  Noise,”  section  3.7,  B.  S.T.  J.  Vol.  23, 
pp.  282-332  and  Vol.  24,  pp.  46-156  (1945-6).  Also  available  in  “Selected  Papers  on 
Noise  and  Stochastic  Processes,”  N.  Wax,  Ed.,  pp.  133-294  (Dover  Publications,  1954), 

5.  W.  W.  Peterson.  T.  G.  Birdsall,  and  W.  C,  Fox,  “The  Theory  of  Signal  Detectability,” 
IRE  Trans,  on  Information  Theory,  Vol.  4,  pp.  171-212,  September,  1954. 

6.  P.  M.  Woodward,  “Probability  and  Information  Theory  with  Applications  to  Radar” 
(McGraw-Hill,  1953). 

7.  R,  Manasse,  “Range  and  Velocity  Accuracy  from  Radar  Measurements,”  Lincoln  Labo¬ 
ratory  Memo  of  February,  1955  (reissued  as  Lincoln  Laboratory  Group  Report  312-26). 

8.  R.  Manasse,  “An  Analysis  of  Angular  Accuracies  from  Radar  Measurements,”  Lincoln 
Laboratory  Group  Report  32-24,  6  December,  1955, 


UNCLASSIFIED 


UNCLASSIFIED 


